Search Results: "justin"

17 August 2016

Charles Plessy: Who finished DEP 5?

Many people worked on finishing DEP 5. I think that the blog of Lars does not show enough how collective the effort was. Looking in the specification's text, one finds:

The following alphabetical list is incomplete; please suggest missing people:
Russ Allbery, Ben Finney, Sam Hocevar, Steve Langasek, Charles Plessy, Noah
Slater, Jonas Smedegaard, Lars Wirzenius.

The Policy's changelog mentions:

  * Include the new (optional) copyright format that was drafted as
    DEP-5.  This is not yet a final version; that's expected to come in
    the 3.9.3.0 release.  Thanks to all the DEP-5 contributors and to
    Lars Wirzenius and Charles Plessy for the integration into the
    Policy package.  (Closes: #609160)
 -- Russ Allbery <rra@debian.org>  Wed, 06 Apr 2011 22:48:55 -0700

and

debian-policy (3.9.3.0) unstable; urgency=low
  [ Russ Allbery ]
  * Update the copyright format document to the version of DEP-5 from the
    DEP web site and apply additional changes from subsequent discussion
    in debian-devel and debian-project.  Revise for clarity, to add more
    examples, and to update the GFDL license versions.  Thanks, Steve
    Langasek, Charles Plessy, Justin B Rye, and Jonathan Nieder.
    (Closes: #658209, #648387)

On my side, I am very grateful to Bill Alombert for having committed the document in the Git repository, which ended the debates.

30 March 2016

Colin Watson: Re-signing PPAs

Julian has written about their efforts to strengthen security in APT, and shortly before that notified us that Launchpad s signatures on PPAs use weak SHA-1 digests. Unfortunately we hadn t noticed that before; GnuPG s defaults tend to result in weak digests unless carefully tweaked, which is a shame. I started on the necessary fixes for this immediately we heard of the problem, but it s taken a little while to get everything in place, and I thought I d explain why since some of the problems uncovered are interesting in their own right. Firstly, there was the relatively trivial matter of using SHA-512 digests on new signatures. This was mostly a matter of adjusting our configuration, although writing the test was a bit tricky since PyGPGME isn t as helpful as it could be. (Simpler repository implementations that call gpg from the command line should probably just add the --digest-algo SHA512 option instead of imitating this.) After getting that in place, any change to a suite in a PPA will result in it being re-signed with SHA-512, which is good as far as it goes, but we also want to re-sign PPAs that haven t been modified. Launchpad hosts more than 50000 active PPAs, though, a significant percentage of which include packages for sufficiently recent Ubuntu releases that we d want to re-sign them for this. We can t expect everyone to push new uploads, and we need to run this through at least some part of our usual publication machinery rather than just writing a hacky shell script to do the job (which would have no idea which keys to sign with, to start with); but forcing full reprocessing of all those PPAs would take a prohibitively long time, and at the moment we need to interrupt normal PPA publication to do this kind of work. I therefore had to spend some quality time working out how to make things go fast enough. The first couple of changes (1, 2) were to add options to our publisher script to let us run just the one step we need in careful mode: that is, forcibly re-run the Release file processing step even if it thinks nothing has changed, and entirely disable the other steps such as generating Packages and Sources files. Then last week I finally got around to timing things on one of our staging systems so that we could estimate how long a full run would take. It was taking a little over two seconds per archive, which meant that if we were to re-sign all published PPAs then that would take more than 33 hours! Obviously this wasn t viable; even just re-signing xenial would be prohibitively slow. The next question was where all that time was going. I thought perhaps that the actual signing might be slow for some reason, but it was taking about half a second per archive: not great, but not enough to account for most of the slowness. The main part of the delay was in fact when we committed the database transaction after processing each archive, but not in the actual PostgreSQL commit, rather in the ORM invalidate method called to prepare for a commit. Launchpad uses the excellent Storm for all of its database interactions. One property of this ORM (and possibly of others; I ll cheerfully admit to not having spent much time with other ORMs) is that it uses a WeakValueDictionary to keep track of the objects it s populated with database results. Before it commits a transaction, it iterates over all those alive objects to note that if they re used in future then information needs to be reloaded from the database first. Usually this is a very good thing: it saves us from having to think too hard about data consistency at the application layer. But in this case, one of the things we did at the start of the publisher script was:

def getPPAs(self, distribution):
    """Find private package archives for the selected distribution."""
    if (self.isCareful(self.options.careful_publishing) or
            self.options.include_non_pending):
        return distribution.getAllPPAs()
    else:
        return distribution.getPendingPublicationPPAs()
def getTargetArchives(self, distribution):
    """Find the archive(s) selected by the script's options."""
    if self.options.partner:
        return [distribution.getArchiveByComponent('partner')]
    elif self.options.ppa:
        return filter(is_ppa_public, self.getPPAs(distribution))
    elif self.options.private_ppa:
        return filter(is_ppa_private, self.getPPAs(distribution))
    elif self.options.copy_archive:
        return self.getCopyArchives(distribution)
    else:
        return [distribution.main_archive]

That innocuous-looking filter means that we do all the public/private filtering of PPAs up-front and return a list of all the PPAs we intend to operate on. This means that all those objects are alive as far as Storm is concerned and need to be considered for invalidation on every commit, and the time required for that stacks up when many thousands of objects are involved: this is essentially accidentally quadratic behaviour, because all archives are considered when committing changes to each archive in turn. Normally this isn t too bad because only a few hundred PPAs need to be processed in any given run; but if we re running in a mode where we re processing all PPAs rather than just ones that are pending publication, then suddenly this balloons to the point where it takes a couple of seconds. The fix is very simple, using an iterator instead so that we don t need to keep all the objects alive:

from itertools import ifilter
def getTargetArchives(self, distribution):
    """Find the archive(s) selected by the script's options."""
    if self.options.partner:
        return [distribution.getArchiveByComponent('partner')]
    elif self.options.ppa:
        return ifilter(is_ppa_public, self.getPPAs(distribution))
    elif self.options.private_ppa:
        return ifilter(is_ppa_private, self.getPPAs(distribution))
    elif self.options.copy_archive:
        return self.getCopyArchives(distribution)
    else:
        return [distribution.main_archive]

After that, I turned to that half a second for signing. A good chunk of that was accounted for by the signContent method taking a fingerprint rather than a key, despite the fact that we normally already had the key in hand; this caused us to have to ask GPGME to reload the key, which requires two subprocess calls. Converting this to take a key rather than a fingerprint gets the per-archive time down to about a quarter of a second on our staging system, about eight times faster than where we started. Using this, we ve now re-signed all xenial Release files in PPAs using SHA-512 digests. On production, this took about 80 minutes to iterate over around 70000 archives, of which 1761 were modified. Most of the time appears to have been spent skipping over unmodified archives; even a few hundredths of a second per archive adds up quickly there. The remaining time comes out to around 0.4 seconds per modified archive. There s certainly still room for speeding this up a bit. We wouldn t want to do this procedure every day, but it s acceptable for occasional tasks like this. I expect that we ll similarly re-sign wily, vivid, and trusty Release files soon in the same way.

18 March 2016

Enrico Zini: Adjusting with Moore's Law

In my last post I used zxcvbn to check the strength of passwords, and it gave estimates in hundreds of centuries. However, there's Moore's law. If we assume that every 18 months the effective computing power will double...

#!/usr/bin/python3
import sys
years = int(sys.argv[1])
# How many 18 months intervals we spent
count = 0
# How many original-computation-time-months we compute each 18 months interval
step = 18
# How many original-computation-time-months we computed so far
total = 0
# How many original-computation-time-months we need
target = years * 12
# Iterate each 18 months interval
while True:
    count += 1
    total += step
    if total > target: break
    step *= 2
print(" :.2f  years needed".format(count / 12 * 18))

...then something that is now estimated in 152 centures is likely to take, well:

$ ./actual_years 15200
9.33 years needed

Update Fixed algorithm (thanks John MacGregor). Now the estimate is a little longer, but still:

$ ./actual_years 15200
21.00 years needed

ukasz Stelmach proposed an analytic solution:

There are two cars (remember phisics in primary school?). They both have the same speed V0. It is going to take "54.697 centuries" for car A to travel the desired distance at constant speed. Car B is accelerating, its speed is 2^(t/18m)*V0 (18 [m]onths). s = V0 * T (car A, T is 54.697 centuries) s = 2^(t/18)*V0 dt (car B, we're looking for t) It's been some time but as far as I can tell it is s = V0 * 2^(t/18) * 1/((1/18) * ln 2) 2^(t/18) * 1/((1/18) * ln 2) = 54.697 * 100 * 12 [months] 2^(t/18) = 2527.54 t = 18*log2(2527.54) t = 16.96 years (still reasonable strength).

6 February 2016

Andrew Shadura: Community time at Collabora

I haven t yet blogged about this (as normally I don t blog often), but I joined Collabora in June last year. Since then, I had an opportunity to work with OpenEmbedded again, write a kernel patch, learn lots of things about systemd (in particular, how to stop worrying about it taking over the world and so on), and do lots of other things. As one would expect when working for a free software consultancy, our customers do understand the value of the community and contributing back to it, and so does the customer for the project I m working on. In fact, our customer insists we keep the number of locally applied patches to, for example, Linux kernel, to minimum, submitting as much as possible upstream. However, apart from the upstreaming work which may be done for the customer, Collabora encourages us, the engineers, to spend up to two hours weekly for upstreaming on top of what customers need, and up to five days yearly as paid Community days. These community days may be spent working on the code or doing volunteering at free software events or even speaking at conferences. Even though on this project I have already been paid for contributing to the free software project which I maintained in my free time previously (ifupdown), paid community time is a great opportunity to contribute to the projects I m interested in, and if the projects I m interested in coincide with the projects I m working with, I effectively can spend even more time on them. A bit unfortunately for me, I haven t spent enough time last year to plan my community days, so I used most of them in the last weeks of the calendar year, and I used them (and some of my upstreaming hours) on something that benefitted both free software community and Collabora. I m talking about SparkleShare, a cross-platform Git-based file synchronisation solution written in C#. SparkleShare provides an easy to use interface for Git, or, actually, it makes it possible to not use any Git interface at all, as it monitors the working directory using inotify and commits stuff right after it changes. It automatically handles conflicts even for binary files, even though I have to admit its handling could still be improved.

At Collabora, we use SparkleShare to store all sorts of internal documents, and it s being used by users not familiar with command line interfaces too. Unfortunately, the version we recently had in Debian had a couple of very annoying bugs, making it a great pain to use it: it would not notice edits in local files, or not notice new commits being pushed to the server, and that led to individual users edits being lost sometimes. Not cool, especially when the document has to be sent to the customer in a couple of minutes. The new versions, 1.4 (and recently released 1.5) was reported as being much better and also fixing some crashes, but it also used GTK+ 3 and some libraries not yet packaged for Debian. Thanh Tung Nguyen packaged these packages (and a newer SparkleShare) for Ubuntu and published them in his PPA, but they required some work to be fit for Debian. I have never touched Mono packages before in my life, so I had to learn a lot. Some time was spent talking to upstream about fixing their copyright statements (they had none in the code, and only one author was mentioned in configure.ac, and nowhere else in the source), a bit more time went into adjusting and updating the patches to the current source code version. Then, of course, waiting the packages to go through NEW. Fixing parallel build issues, waiting for buildds to all build dependencies for at least one architecture But then, finally, on 19^th of January I had the updated SparkleShare in Debian. As you may have already guessed, this blog post has been sponsored by Collabora, the first of my employers to ~~encourage~~ require me to work on free software in my paid time :)

29 January 2016

Jan Wagner: Oxidized - silly attempt at (Really Awesome New Cisco confIg Differ)

Since ages I wanted have replaced this freaking backup solution of our Network Equipment based on some hacky shell scripts and expect uploading the configs on a TFTP server. Years ago I stumbled upon RANCID (Really Awesome New Cisco confIg Differ) but had no time to implement it. Now I returned to my idea to get rid of all our old crap.
I don't know where, I think it was at DENOG2, I saw RANCID coupled with a VCS, where the NOC was notified about configuration (and inventory) changes by mailing the configuration diff and the history was indeed in the VCS.
The good old RANCID seems not to support to write into a VCS out of the box. But for the rescue there is rancid-git, a fork that promises git extensions and support for colorized emails. So far so good. While I was searching for a VCS capable RANCID, somewhere under a stone I found Oxidized, a 'silly attempt at rancid'. Looking at it, it seems more sophisticated, so I thought this might be the right attempt. Unfortunately there is no Debian package available, but I found an ITP created by Jonas. Anyway, for just looking into it, I thought the Docker path for a testbed might be a good idea, as no Debian package ist available (yet). For oxidized configuration is only a configfile needed and as nodes source a rancid compatible router.db file can be used (beside SQLite and http backend). A migration into a production environment seems pretty easy. So I gave it a go. I assume Docker is installed already. There seems to be a Docker image on Docker Hub, that looks official, but it seems not maintained (actually). An issue is open for automated building the image.

Creating Oxidized container image The official documentation describes the procedure. I used a slightly different approach.

docking-station:~# mkdir -p /srv/docker/oxidized/  
docking-station:~# git clone https://github.com/ytti/oxidized \  
 /srv/docker/oxidized/oxidized.git
docking-station:~# docker build -q -t oxidized/oxidized:latest \  
 /srv/docker/oxidized/oxidized.git

I thought it might be a good idea to also tag the image with the actual version of the gem.

docking-station:~# docker tag oxidized/oxidized:latest \  
 oxidized/oxidized:0.11.0
docking-station:~# docker images   grep oxidized  
oxidized/oxidized   latest    35a325792078  15 seconds ago  496.1 MB  
oxidized/oxidized   0.11.0    35a325792078  15 seconds ago  496.1 MB

Create initial default configuration like described in the documentation.

docking-station:~# mkir -p /srv/docker/oxidized/.config/  
docking-station:~# docker run -e CONFIG_RELOAD_INTERVAL=600 \  
 -v /srv/docker/oxidized/.config/:/root/.config/oxidized \
 -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized

Adjusting configuration After this I adjusted the default configuration for writing a log, the nodes config into a bare git, having nodes secrets in router.db and some hooks for debugging.

Creating node configuration

docking-station:~# echo "7204vxr.lab.cyconet.org:cisco:admin:password:enable" >> \  
 /srv/docker/oxidized/.config/router.db
docking-station:~# echo "ccr1036.lab.cyconet.org:routeros:admin:password" >> \  
 /srv/docker/oxidized/.config/router.db

Starting the oxidized beast

docking-station:~# docker run -e CONFIG_RELOAD_INTERVAL=600 \  
 -v /srv/docker/oxidized/.config/:/root/.config/oxidized \
 -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized
Puma 2.16.0 starting...  
* Min threads: 0, max threads: 16
* Environment: development
* Listening on tcp://127.0.0.1:8888

If you want to have the container get started with the docker daemon automatically, you can start the container with --restart always and docker will take care of it. If I wanted to make it running permanent, I would use a systemd unitfile.

Reload configuration immediately If you don't want to wait to automatically reload of the configuration, you can trigger it.

docking-station:~# curl -s http://localhost:8888/reload?format=json \  
 -O /dev/null
docking-station:~# tail -2 /srv/docker/oxidized/.config/log/oxidized.log  
I, [2016-01-29T16:50:46.971904 #1]  INFO -- : Oxidized starting, running as pid 1  
I, [2016-01-29T16:50:47.073307 #1]  INFO -- : Loaded 2 nodes

Writing nodes configuration

docking-station:/srv/docker/oxidized/.config/oxidized.git# git shortlog  
Oxidizied (2):  
      update 7204vxr.lab.cyconet.org
      update ccr1036.lab.cyconet.org

Writing the nodes configurations into a local bare git repository is neat but far from perfect. It would be cool to have all the stuff in a central VCS. So I'm pushing it every 5 minutes into one with a cron job.

docking-station:~# cat /etc/cron.d/doxidized  
# m h dom mon dow user  command                                                 
*/5 * * * *    root    $(/srv/docker/oxidized/bin/oxidized_cron_git_push.sh)
docking-station:~# cat /srv/docker/oxidized/bin/oxidized_cron_git_push.sh  
#!/bin/bash
DOCKER_OXIDIZED_BASE="/srv/docker/oxidized/"  
OXIDIZED_GIT_DIR=".config/oxidized.git"
cd $ DOCKER_OXIDIZED_BASE /$ OXIDIZED_GIT_DIR   
git push origin master --quiet

Now having all the nodes configurations in a source code hosting system, we can browse the configurations, changes, history and even establish notifications for changes. Mission accomplished! Now I can test the coverage of our equipment. The last thing that would make me super happy, a oxidized Debian package!

26 December 2015

Russ Allbery: kstart 4.2

kstart is a collection of two utilities for managing Kerberos tickets, particularly focused on the needs of long-running commands or daemons. This release has taken embarassingly long to release, given that the last release was in January of 2012. Sorry about that, particularly those who were stuck on various bugs! This code base, particularly the test suite, is in dire need of significant refactoring, which reduces my desire to work on it, and then I ran very, very short on free software time. This release is important because I think I've finally worked through all the various tricky complexity of startup when running a command under k5start. The core problem: the original k5start exited when authentication failed. However, it's very nice to start daemons that need a Kerberos ticket by running them via k5start. But if this is done during system startup, the network may not be up yet, and one may not be able to contact the KDC. It's very difficult to get the timing right just by adjusting init scripts. The core fix in this release is that k5start run as a daemon or with a command, and krenew run with the -i option, now keeps retrying the authentication or renewal until it succeeds. It's retried immediately and then with exponential backoff, starting with a delay of a second, until intervals of one minute, and then retried at a delay of one minute until it succeeds. Any command isn't run until the authentication works, so (unlike previous versions) k5start won't start the command without a Kerberos ticket. That should fix various problems people are having. There are also some new options to tweak the renewal behavior. -H can now be used in combination with -K to specify how long of a minimum ticket lifetime k5start or krenew should try to maintain. Alternately, there's now a -a flag that tells both to attempt an authentication or renewal every time they wake up (at the interval specified with -K). This probably should have been the default for -K originally, but I didn't change the original behavior for backwards compatibility. There are also a couple of bug fixes: the temporary ticket cache created by k5start is cleaned up properly, and the programs no longer incorrectly reject the combination of -b and -K or a command. You can get the latest version from the kstart distribution page.

14 August 2015

Wouter Verhelst: Multi-pass transcoding to WebM with normalisation

Transcoding video from one format to another seems to be a bit of a black art. There are many tools that allow doing this kind of stuff, but one issue that most seem to have is that they're not very well documented. I ran against this a few years ago, when I was first doing video work for FOSDEM and did not yet have proper tools to do the review and transcoding workflow. At the time, I just used mplayer to look at the .dv files, and wrote a text file with a simple structure to remember exactly what to do with it. That file was then fed to a perl script which wrote out a shell script that would use the avconv command to combine and extract the "interesting" data from the source DV files into a single DV file per talk, and which would then call a shell script which used gst-launch and sox to do a multi-pass transcode of those intermediate DV files into a WebM file. While all that worked properly, it was a rather ugly hack, never cleaned up, and therefore I never really documented it properly either. Recently, however, someone asked me to do so anyway, so here goes. Before you want to complain about how this ate the videos of your firstborn child, however, note the above. The perl script spent a somewhat large amount of code reading out the text file and parsing it into an array of hashes. I'm not going to reproduce that, since the actual format of the file isn't all that important anyway. However, here's the interesting bits:

foreach my $pfile(keys %parts)  
        my @files = @ $parts $pfile ;
        say "#" x (length($pfile) + 4);
        say "# " . $pfile . " #";
        say "#" x (length($pfile) + 4);
        foreach my $file(@files)  
                my $start = "";
                my $stop = "";
                if(defined($file-> start ))  
                        $start = "-ss " . $file-> start ;
                 
                if(defined($file-> stop ))  
                        $stop = "-t " . $file-> stop ;
                 
                if(defined($file-> start ) && defined($file-> stop ))  
                        my @itime = split /:/, $file-> start ;
                        my @otime = split /:/, $file-> stop ;
                        $otime[0]-=$itime[0];
                        $otime[1]-=$itime[1];
                        if($otime[1]<0)  
                                $otime[0]-=1;
                                $otime[1]+=60;
                         
                        $otime[2]-=$itime[2];
                        if($otime[2]<0)  
                                $otime[1]-=1;
                                $otime[2]+=60;
                         
                        $stop = "-t " . $otime[0] . ":" . $otime[1] .  ":" . $otime[2];
                 
                if(defined($file-> start )   defined($file-> stop ))  
                        say "ln " . $file-> name  . ".dv part-pre.dv";
                        say "avconv -i part-pre.dv $start $stop -y -acodec copy -vcodec copy part.dv";
                        say "rm -f part-pre.dv";
                  else  
                        say "ln " . $file-> name  . ".dv part.dv";
                 
                say "cat part.dv >> /tmp/" . $pfile . ".dv";
                say "rm -f part.dv";
         
        say "dv2webm /tmp/" . $pfile . ".dv";
        say "rm -f /tmp/" . $pfile . ".dv";
        say "scp /tmp/" . $pfile . ".webm video.fosdem.org:$uploadpath   true";
        say "mv /tmp/" . $pfile . ".webm .";

That script uses avconv to read one or more .dv files and transcode them into a single .dv file with all the start- or end-junk removed. It uses /tmp rather than the working directory, since the working directory was somewhere on the network, and if you're going to write several gigabytes of data to an intermediate file, it's usually a good idea to write them to a local filesystem rather than to a networked one. Pretty boring. It finally calls dv2webm on the resulting .dv file. That script looks like this:

#!/bin/bash
set -e
newfile=$(basename $1 .dv).webm
wavfile=$(basename $1 .dv).wav
wavfile=$(readlink -f $wavfile)
normalfile=$(basename $1 .dv)-normal.wav
normalfile=$(readlink -f $normalfile)
oldfile=$(readlink -f $1)
echo -e "\033]0;Pass 1: $newfile\007"
gst-launch-0.10 webmmux name=mux ! fakesink \
  uridecodebin uri=file://$oldfile name=demux \
  demux. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=1 threads=2 ! queue ! mux.video_0 \
  demux. ! progressreport ! audioconvert ! audiorate ! tee name=t ! queue ! vorbisenc ! queue ! mux.audio_0 \
  t. ! queue ! wavenc ! filesink location=$wavfile
echo -e "\033]0;Audio normalize: $newfile\007"
sox --norm $wavfile $normalfile
echo -e "\033]0;Pass 2: $newfile\007"
gst-launch-0.10 webmmux name=mux ! filesink location=$newfile \
  uridecodebin uri=file://$oldfile name=video \
  uridecodebin uri=file://$normalfile name=audio \
  video. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=2 threads=2 ! queue ! mux.video_0 \
  audio. ! progressreport ! audioconvert ! audiorate ! vorbisenc ! queue ! mux.audio_0
rm $wavfile $normalfile

... and is a bit more involved. Multi-pass encoding of video means that we ask the encoder to first encode the file but store some statistics into a temporary file (/tmp/vp8-multipass, in our script), which the second pass can then reuse to optimize the transcoding. Since DV uses different ways of encoding things than does VP8, we also need to do a color space conversion (ffmpegcolorspace) and deinterlacing (deinterlace), but beyond that the video line in the first gstreamer pipeline isn't very complicated. Since we're going over the file anyway and we need the audio data for sox, we add a tee plugin at an appropriate place in the audio line in the first gstreamer pipeline, so that we can later on pick up that same audio data an write it to a wav file containing linear PCM data. Beyond the tee, we go on and do a vorbis encoding, as is needed for the WebM format. This is not actually required for a first pass, but ah well. There's some more conversion plugins in the pipeline (specifically, audioconvert and audiorate), but those are not very important. We next run sox --norm on the .wav file, which does a fully automated audio normalisation on the input. Audio normalisation is the process of adjusting volume levels so that the audio is not too loud, but also not too quiet. Sox has pretty good support for this; the default settings of its --norm parameter make it adjust the volume levels so that the highest peak will just about reach the highest value that the output format can express. As such, you have no clipping anywhere in the file, but also have an audio level that is actually useful. Next, we run a second-pass encoding on the input file. This second pass uses the statistics gathered in the first pass to decide where to put its I- and P-frames so that they are placed at the most optimal position. In addition, rather than reading the audio from the original file, we now read the audio from the .wav file containing the normalized audio which we produced with sox, ensuring the audio can be understood. Finally, we remove the intermediate audio files we created; and the shell script which was generated by perl also contained an rm command for the intermediate .dv file. Some of this is pretty horrid, and I never managed to clean it up enough so it would be pretty (and now is not really the time). However, it Just Works(tm), and I am happy to report that it continues to work with gstreamer 1.0, provided you replace the ffmpegcolorspace by an equally simple videoconvert, which performs what ffmpegcolorspace used to perform in gstreamer 0.10.

12 August 2015

Daniel Pocock: Recording live events like a pro (part 2: video)

In the first blog in this series, we looked at how to capture audio effectively for a range of different events. While recording audio appears less complicated than video, it remains fundamental to a good recording. For some types of event, like a speech or a debate, you can have quite a bad video with poor lighting and other defects but people will still be able to watch it if the audio is good. Therefore, if you haven't already looked at the previous blog, please do so now. As mentioned in the earlier blog, many people now have high quality equipment for recording both audio and video and a wide range of opportunities to use it, whether it is a talk at a conference, a wedding or making a Kickstarter video. The right camera for video recording The fundamental piece of equipment is the camera itself. You may have a DSLR camera that can record video or maybe you have a proper handheld video camera. The leading DSLR cameras, combined with a good lens, make higher quality recordings than many handheld video cameras. Unfortunately, although you pay good money to buy a very well engineered DSLR that could easily record many hours of video, most DSLRs are crippled to record a maximum of 30 minutes in one recording. This issue and some workarounds are discussed later in this blog. If you don't have any camera at all you need to think carefully about which type to buy. If you are only going to use it once you may want to consider renting or borrowing or asking for other people attending the event to bring whatever cameras they have to help make multiple recordings (the crowdsourcing solution). If you are a very keen photographer then you will probably have a preference for a DSLR. Accessories Don't just look at the cost of your camera and conclude that is all the budget you need. For professional quality video recording, you will almost certainly need some accessories. You may find they are overpriced at the retail store where you bought your camera, but you still need some of them, so have a look online.

Recording a talk at a free software event with a Nikon D800 on a very basic tripod with Rode VideoMic Pro, headphones (white cable) and external power (black cable) If you want to capture audio with the camera and record it in the video file (discussed in more detail below), you will need to purchase a microphone that mounts on the camera. The built-in microphones on cameras are often quite bad, even on the most expensive cameras. If you are just using the built-in microphone for reference audio (to help with time synchronization when you combine different audio files with the video later) then the built-in microphone may be acceptable. Camera audio is discussed in more detail below. If your camera has a headphone socket, get some headphones for listening to the audio. Make sure you have multiple memory cards. Look carefully at the speed of the memory cards, slow ones are cheaper but they can't keep up with the speed of writing 1080p video. At a minimum, you should aim to buy memory cards that can handle one or two days worth of data for whatever it is you do. A tripod is essential for most types of video. If you use a particularly heavy camera or lens or if you are outdoors and it may be windy you will need a heavier tripod for stability. For video, it is very useful to have a tripod with a handle for panning left and right but if the camera will be stationary for the whole recording then the handle is not essential. Carrying at least one spare battery is another smart move. On one visit to the Inca Trail in Peru, we observed another member of our group hiking up and down the Andes with a brand new DSLR that they couldn't use because the battery was flat. For extended periods of recording, batteries will not be sufficient and you will need to purchase a mains power supply (PSU). These are available for most types of DSLR and video camera. The camera vendors typically design cameras with unusual power sockets so that you can only use a very specific and heavily overpriced PSU from the same company. Don't forget a surge protector too. There are various smartphone apps that allow you to remotely control the camera from the screen of your phone, such as the qDslrDashboard app. These often give a better preview than the screen built-in to the camera and may even allow you to use the touch screen to focus more quickly on a specific part of the picture. A regular USB cable is not suitable for this type of app, you need to buy a USB On-The-Go (OTG) cable.

Screenshot of qDslrDashboard app on a smartphone, controlling a DSLR camera If you plan to copy the video from the camera to a computer at the event, you will need to make sure you have a fast memory card reader. The memory card readers in some laptops are quite slow and others can be quite fast so you may not need to buy an extra card reader. Camera audio Most cameras, including DSLRs, have a built-in microphone and a socket for connecting an external microphone. The built-in microphones obtain very poor quality sound. For many events, it is much better to have independent microphones, such as a lapel microphone attached to a smartphone or wireless transmitter. Those solutions are described in part one of this blog series. Nonetheless, there are still some benefits of capturing audio in the camera. The biggest benefit is the time synchronization: if you have audio recordings in other devices, you will need to align them with the video using post-production software. If the camera recorded an audio stream too, even if the quality is not very good, you can visualize the waveform on screen and use it to align the other audio recordings much more easily and precisely. If the camera will be very close to the people speaking then it may be acceptable to use a microphone mounted on the camera. This will be convenient for post-production because the audio will be synchronized with the video. It may still not be as good as a lapel microphone though, but the quality of these camera-mounted microphones is still far higher than the built-in microphones. I've been trying the Rode VideoMic Pro, it is definitely better than recording with the built-in microphone on the camera and also better than the built-in microphone on my phone. One problem that most people encounter is the sound of the lens autofocus mechanism being detected by the microphone. This occurs with both the built-in microphone and any other microphone you mount on the camera. A microphone mounted on top of the camera doesn't detect this noise with the same intensity as the built-in microphone but it is still present in the recordings. If using a camera-mounted microphone to detect the audio from an audience, you may need to have an omnidirectional microphone. Many camera-mounted microphones are directional and will not detect very much sound from the sides or behind the camera. When using any type of external microphone with the camera, it is recommend to disable automatic gain control (AGC) in the camera settings and then manually adjust the microphone sensitivity/volume level. Use headphones A final word on audio - most good cameras have an audio output socket. Connect headphones and wear them, to make sure you are always capturing audio. Otherwise, if the microphone's battery goes flat or if a wireless microphone goes out of range you may not notice. Choosing a lens The more light you get, the better. Bigger and more expensive lenses allow more light into the camera. Many of the normal lenses sold with a DSLR camera are acceptable but if it is a special occasion you may want to rent a more expensive lens for the day. If you already have a lens, it is a very good idea to test it in conditions similar to those you expect for the event you want to record. Recording duration limits Most DSLR cameras with video capability impose a 30 minute maximum recording duration This is basically the result of a friendly arrangement between movie studios and politicians to charge an extra tax on video recording technology and potentially make the movie studio bosses richer, supposedly justified by the fact that a tiny but exaggerated number of people use their cameras to record movies at the cinema. As a consequence, most DSLR manufacturers limit the duration of video recording so their product won't be subject to the tax, ensuring the retail price is lower and more attractive. On top of this, many DSLR cameras also have a 4GB file size limit if they use the FAT filesystem. Recording 1080p video at a high frame rate may hit the file size limit in 10 minutes, well before you encounter the 30 minute maximum recording duration. To deal with the file size issue, you can record at 720p instead of 1080p and use the frame rate 24fps. For events longer than 30 minutes or where you really want 1080p or a higher frame rate, there are some other options you can consider:

Buy or rent a proper video camera instead of using a DSLR camera
Using multiple cameras that can stop and be restarted at different times.
Manually stopping and restarting the camera if there are breaks in the event where it is safe to do so.
Use an app to control the camera and program it to immediately restart the recording each time it stops
Extract the raw output from the camera's HDMI socket and record into some other device or computer. There are several purpose-built devices
that can be used this way with an embedded SSD for storage.
There are also some people distributing unofficial/alternative firmware images that remove the artificial 30 minute recording limit.

Camera settings There are many online tutorials and demonstration videos on YouTube that will help you optimize the camera settings for video. You may have already made recordings using the automatic mode. Adjusting some or all of the settings manually may help you create a more optimal recording. You will need to spend some time familiarizing yourself with the settings first. The first thing to check is white balance. This tells the camera the type of lighting in the location. If you set this incorrectly then the colours will be distorted. Many cameras have the ability to automatically set the white balance. For video, you may be able to change one or more settings that control the recording quality. These settings control the file compression ratio and image size. Typical image size settings are 720p and 1080p. Compression ratio may be controlled by a high/medium/low quality setting. Choosing the highest quality and biggest picture requires more space on the memory card and also means you reach the 4GB file size limit more quickly. A higher quality setting also implies a faster memory card is required, because the rate of megabytes per second written to the memory card is higher. Next you need to think about the frame rate. Events that involve fast moving subjects, such as sports, typically benefit from a higher frame rate. For other events it is quite acceptable to use 24 frames per second (fps). Higher frame rates also imply bigger file size and a requirement for a faster memory card. Once you have decided on the frame rate, the next thing to do is set the shutter speed. Use a shutter speed that is double the frame rate. For example, if using 24fps or 25fps, use a 1/50 shutter speed. The final two settings you need to adjust are the ISO and aperture. Set these based on the lighting conditions and extent to which the subjects are moving. For example, if the setting is dark or if you are trying to record fast moving subjects like athletes, vehicles or animals, use an ISO value of 800 or higher. Once you have chosen ISO, adjust the aperture to ensure the picture is sufficiently illuminated. Aperture also has a significant impact on the depth of field. Operating the camera: zoom and focus Many people use zoom lenses. It is not always necessary to change the zoom while recording a video, you can use software to zoom in and out on individual parts of the picture when editing it in post-production. If you do change the zoom while recording, it may be more difficult to maintain focus. Almost all lenses support manual focus (turning the focus ring by hand) and many support automatic focus. When shooting photographs with a DSLR, the mirror is down and the camera can use dedicated sensors for focus and light sensing. When shooting video, the mirror is up and the camera can not use the same focus sensors that are used in photography. Video recording uses a digital focussing algorithm based on contrast in the picture. If you take a lot of photos you are probably quite accustomed to the fast and precise autofocus for photography and you will notice that keeping a video in focus is more challenging. As mentioned already, one of the first things you can do to keep focus simple is to avoid zooming while recording. Record in a higher resolution than you require and then zoom with software later. Some people record using 4k resolution even when they only want to produce a 720p video, as they can digitally zoom in to different parts of the 4k recording without losing detail. If the subject is very stationary (people sitting at a desk for an interview is a typical example), you may be able to set the camera to manual focus and not change it at all while recording. If you choose to enable autofocus while recording, any built-in camera microphone or microphone mounted on the camera is likely to detect sounds from the motorized focus system. Ultimately, the autofocus mechanism is not accurate for all subjects and you may be unable to stop them moving around so you will need to change the focus manually while recording. It requires some practice to be able to do this quickly without overshooting the right focus. To make life more tricky, Nikon and Canon focus rings rotate in the opposite direction, so if you are proficient using one brand you may feel awkward if you ever have to use the other. A good way to practice this skill is to practice while in the car or on the train, pointing at different subjects outside the window and trying to stay in focus as you move from one subject to the next. Make a trial run Many events, from weddings right up to the Olympic Games opening ceremony, have a trial run the day before. One reason for that is to test the locations and settings of all the recording and broadcasting equipment. If a trial run isn't possible for your event, you may find some similar event to practice recording and test your equipment. For example, if you are planning to record a wedding, you could try and record a Sunday mass in the same church. Backup and duplicate the recordings before leaving the event If you only have one copy of the recordings and the equipment is stolen or damaged you may be very disappointed. Before your event, make a plan to duplicate the raw audio and video recordings so that several people can take copies away with them. Decide in advance who will be responsible for this, ensure there will be several portable hard disks and estimate how much time it will take to prepare the copies and factor this into the schedule. Conclusion All the products described can be easily purchased from online retailers. You may not need every accessory that is mentioned as it depends on the type of event you record. The total cost of buying or renting the necessary accessories may be as much as the cost of the camera itself so if you are new to this you may need to think carefully about making a budget with a spreadsheet to do it correctly. Becoming familiar with the camera controls and practicing the techniques for manual focus and zoom can take weeks or months. If you enjoy photography this can be time well spent but if you don't enjoy it then you may not want to commit the time necessary to make good quality video. Don't rule out options like renting equipment instead of buying it or crowdsourcing, asking several participants or friends to help make recordings with their own equipment. For many events, audio is far more indispensable than video and as emphasizing at the beginning of this article, it is recommended that you should be one hundred percent confident in your strategy for recording audio before you start planning to record video.

2 July 2015

Petter Reinholdtsen: MakerCon Nordic videos now available on Frikanalen

Last oktober I was involved on behalf of NUUG with recording the talks at MakerCon Nordic, a conference for the Maker movement. Since then it has been the plan to publish the recordings on Frikanalen, which finally happened the last few days. A few talks are missing because the speakers asked the organizers to not publish them, but most of the talks are available. The talks are being broadcasted on RiksTV channel 50 and using multicast on Uninett, as well as being available from the Frikanalen web site. The unedited recordings are available on Youtube too. This is the list of talks available at the moment. Visit the Frikanalen video pages to view them.

Evolutionary algorithms as a design tool - from art to robotics (Kyrre Glette)
Make and break (Hans Gerhard Meier)
Making a one year school course for young makers (Olav Helland)
Innovation Inspiration - IPR Databases as a Source of Inspiration (Hege Langlo)
Making a toy for makers (Erik Torstensson)
How to make 3D printer electronics (Elias Bakken)
Hovering Clouds: Looking at online tool offerings for Product Design and 3D Printing (William Kempton)
Travelling maker stories ( yvind Nydal Dahl)
Making the first Maker Faire in Sweden (Nils Olander)
Breaking the mold: Printing 1000 s of parts (Espen Sivertsen)
Ultimaker and open source 3D printing (Erik de Bruijn)
Autodesk s 3D Printing Platform: Sparking innovation (Hilde Sevens)
How Making is Changing the World and How You Can Too! (Jennifer Turliuk)
Open-Source Adventuring: OpenROV, OpenExplorer and the Future of Connected Exploration (David Lang)
Making in Norway (Haakon Karlsen Jr., Graham Hayward and Jens Dyvik)
The Impact of the Maker Movement (Mike Senese)

Part of the reason this took so long was that the scripts NUUG had to prepare a recording for publication were five years old and no longer worked with the current video processing tools (command line argument changes). In addition, we needed better audio normalization, which sent me on a detour to package bs1770gain for Debian. Now this is in place and it became a lot easier to publish NUUG videos on Frikanalen.

11 June 2015

Petter Reinholdtsen: Measuring and adjusting the loudness of a TV channel using bs1770gain

Television loudness is the source of frustration for viewers everywhere. Some channels are very load, others are less loud, and ads tend to shout very high to get the attention of the viewers, and the viewers do not like this. This fact is well known to the TV channels. See for example the BBC white paper "Terminology for loudness and level dBTP, LU, and all that" from 2011 for a summary of the problem domain. To better address the need for even loadness, the TV channels got together several years ago to agree on a new way to measure loudness in digital files as one step in standardizing loudness. From this came the ITU-R standard BS.1770, "Algorithms to measure audio programme loudness and true-peak audio level". The ITU-R BS.1770 specification describe an algorithm to measure loadness in LUFS (Loudness Units, referenced to Full Scale). But having a way to measure is not enough. To get the same loudness across TV channels, one also need to decide which value to standardize on. For European TV channels, this was done in the EBU Recommondaton R128, "Loudness normalisation and permitted maximum level of audio signals", which specifies a recommended level of -23 LUFS. In Norway, I have been told that NRK, TV2, MTG and SBS have decided among themselves to follow the R128 recommondation for playout from 2016-03-01. There are free software available to measure and adjust the loudness level using the LUFS. In Debian, I am aware of a library named libebur128 able to measure the loudness and since yesterday morning a new binary named bs1770gain capable of both measuring and adjusting was uploaded and is waiting for NEW processing. I plan to maintain the latter in Debian under the Debian multimedia umbrella. The free software based TV channel I am involved in, Frikanalen, plan to follow the R128 recommondation ourself as soon as we can adjust the software to do so, and the bs1770gain tool seem like a good fit for that part of the puzzle to measure loudness on new video uploaded to Frikanalen. Personally, I plan to use bs1770gain to adjust the loudness of videos I upload to Frikanalen on behalf of the NUUG member organisation. The program seem to be able to measure the LUFS value of any media file handled by ffmpeg, but I've only successfully adjusted the LUFS value of WAV files. I suspect it should be able to adjust it for all the formats handled by ffmpeg.

5 March 2015

EvolvisForge blog: I have to give you that one

After seeing what the Wildfly (formerly JBoss AS) and Liferay combo does to /tmp, and somewhat attempting to fix it, I saw JVM_TMP in the Debian tomcat7 init script and thought, oh no, not another one. Is that even safe, what they do here, or is that a possibility to instantly pwn? The net is full of literature for how to obtain temporary files and directories, but there is nothing about how to reliably obtain paths under /tmp or, more generally, directories not just writable for one single user (think the g+w thing that got FusionForge CVE-2013-1423). The scenario here is: I am root, and I want to start something as another user, and pass it a stable path, such as /tmp/liferay. So I can just mkdir /tmp/liferay die; chown thatuser /tmp/liferay and, in the stop process, rm -rf /tmp/liferay, right? (Of course not. Also, bad example, as the liferay thing can also be started as thatuser, and our devs regularily need to do that, the init script is there just for the admin convenience and reboot-safety. But I still am interested if there is a secure way to achieve this.) The tomcat7 scenario is trivial : on That Other Init System , it would just get its private /tmp declared in the .service file, and good is, no more hassle. That s one I have to give you. (No idea if this is actually shipped in jessie. Our production systems run wheezy anyway, so there is not even the slightest bit of temptation. Plus, it would not solve the liferay issue, see above. Still, a point for going into the right direction.) The idea here is the same. It creates a directory on start and tears it down on stop. If there was nothing to do on start, the init script could just use mktemp -d. Heck, maybe it still should, but it would need to note down, and communicate to the stop instance, the actual name used. What a drag This is something I see popping up from time to time. I want to use stable paths for SSH session multiplexing control sockets in my ssh_config(5) file, but have them on tmpfs (Linux) or mfs (BSD) so they get properly removed on reboot. No Unix traditionally has per-user temporary directories that are clean and created after reboot. (Adjusting the paths is trivial once you have them.) Android has it worse, what with not having a world-writable tmp directory, which the shell needs e.g. for here documents; there are two components here, to have a directory the current user can write to, and to know its location. Some fail at the first, some at the second, some at both, and the classic /tmp is not the cure, as we have seen. (But if you ever see mksh erroring out due to lack of write permissions somewhere (including /sqlite_stmt_journals which used to be it) as non-root on Android, or even as root, set TMPDIR to something writable; it s tracked, so the change gets active immediately.)

18 January 2015

Chris Lamb: Adjusting a backing track with SoX

Earlier today I came across some classical sheet music that included a "playalong" CD, just like a regular recording except it omits the solo cello part. After a quick listen it became clear there were two problems:

The recording was made at A=442, rather than the more standard A=440.
The tempi of the movements was not to my taste, either too fast or too slow.

SoX, the "Swiss Army knife of sound processing programs", can easily adjust the latter, but to remedy the former it must be provided with a dimensionless "cent" unit ie. 1/100th of a semitone rather than the 442Hz and 440Hz reference frequencies. First, we calculate the cent difference with:

https://d1icoid1cnixnp.cloudfront.net/yadt/blog.Image/image/original/24.jpeg

Next, we rip the material from the CD:

$ sudo apt-get install ripit flac
[..]
$ ripit --coder 2 --eject --nointeraction
[..]

And finally we adjust the tempo and pitch:

$ apt-get install sox libsox-fmt-mp3
[..]
$ sox 01.flac 01.mp3 pitch -7.85 tempo 1.00 # (Tuning notes)
$ sox 02.flac 02.mp3 pitch -7.85 tempo 0.95 # Too fast!
$ sox 03.flac 03.mp3 pitch -7.85 tempo 1.01 # Close..
$ sox 04.flac 04.mp3 pitch -7.85 tempo 1.03 # Too slow!

(I'm converting to MP3 at the same time it'll be more convenient on my phone.)

2 December 2014

Matthias Klumpp: How to build a cross-distro package with Limba

Disclaimer: Limba is still

in a very early stage of development. Bugs happen, and I give to guarantees on API stability yet. Limba is a very simple cross-distro package installer, utilizing OverlayFS found in recent Linux kernels (>= 3.18). As example I created a small Limba package for one of the Qt5 demo applications, and I would like to share the process of creating Limba packages it s quite simple, and I could use some feedback on how well the resulting packages work on multiple distributions. I assume that you have compiled Limba and installed it how that is done is described in its README file. So, let s start. 1. Prepare your application The cool thing about Limba is that you don t really have to do many changes on your application. There are a few things to pay attention to, though:

Ensure the binaries and data are installed into the right places in the directory hierarchy. Binaries must go to $prefix/bin, for example.
Ensure that configuration can be found under /etc as well as under $prefix/etc

This needs to be done so your application will find its data at runtime. Additionally, you need to write an AppStream metadata file, and find out which stuff your application depends on. 2. Create package metadata & install software 1.1 Basics Now you can create the metadata necessary to build a Limba package. Just run

cd /path/to/my/project
lipkgen make-template

This will create a pkginstall directory, containing a control file and a metainfo.xml file, which can be a symlink to the AppStream metadata, or be new metadata. Now, configure your application with /opt/bundle as install prefix (-DCMAKE_INSTALL_PREFIX=/opt/bundle, prefix=/opt/bundle, etc.) and install it to the pkginstall/inst_target directory. 1.2 Handling dependencies If your software has dependencies on other packages, just get the Limba packages for these dependencies, or build new ones. Then place the resulting IPK packages in the pkginstall/repo directory. Ideally, you should be able to fetch Limba packages which contain the software components directly from their upstream developers. Then, open the pkginstall/control file and adjust the Requires line. The names of the components you depend on match their AppStream-IDs (<id/> tag in the AppStream XML document). Any version-relation (>=, >>, <<, <=, <>) is supported, and specified in brackets after the component-id. The resulting control-file might look like this:

Format-Version: 1.0

Requires: Qt5Core (>= 5.3), Qt5DBus (>= 5.3), libpng12

If the specified dependencies are in the repo/ subdirectory, these packages will get installed automatically, if your application package is installed. Otherwise, Limba depends on the user to install these packages manually there is no interaction with the distribution s package-manager (yet?). 3. Building the package In order to build your package, make sure the content in inst_target/ is up to date, then run

lipkgen build pkginstall/

This will build your package and output it in the pkginstall/ directory. 4. Testing the package You can now test your package, Just run

sudo lipa install package.ipk

Your software should install successfully. If you provided a .desktop file in $prefix/share/applications, you should find your application in your desktop s application-menu. Otherwise, you can run a binary from the command-line, just append the version of your package to the binary name (bash-comletion helps). Alternatively, you can use the runapp command, which lets you run any binary in your bundle/package, which is quite helpful for debugging (since the environment a Limba-installed application is run is different from the one of other applications). Example:

runapp $ component_id -$ version :/bin/binary-name

And that s it!

I used these steps to create a Limba package for the OpenGL Qt5 demo on Tanglu 2 (Bartholomea), and tested it on Kubuntu 15.04 (Vivid) with KDE, as well as on an up-to-date Fedora 21, with GNOME and without any Qt or KDE stuff installed:

I encountered a few obstacles when building the packages, e.g. Qt5 initially didn t find the right QPA plugin that has been fixed by adjusting a config file in the Qt5Gui package. Also, on Fedora, a matching libpng was missing, so I included that as well. You can find the packages at Github, currently (but I am planning to move them to a different place soon). The biggest issue with Limba is at time, that it needs Linux 3.18, or an older kernel with OverlayFS support compiled in. Apart from that and a few bugs, the experience is quite smooth. As soon as I am sure there are now hidden fundamental issues, I can think of implementing more features, like signing packages and automatically updating them. Have fun playing around with Limba!

30 November 2014

Johannes Schauer: simple email setup

I was unable to find a good place that describes how to create a simple self-hosted email setup. The most surprising discovery was, how much already works after:

apt-get install postfix dovecot-imapd

Right after having finished the installation I was able to receive email (but only in in /var/mail in mbox format) and send email (bot not from any other host). So while I expected a pretty complex setup, it turned out to boil down to just adjusting some configuration parameters.

Postfix The two interesting files to configure postfix are `/etc/postfix/main.cf` and `/etc/postfix/master.cf`. A commented version of the former exists in `/usr/share/postfix/main.cf.dist`. Alternatively, there is the ~600k word strong man page postconf(5). The latter file is documented in master(5).

/etc/postfix/main.cf I changed the following in my main.cf

@@ -37,3 +37,9 @@
 mailbox_size_limit = 0
 recipient_delimiter = +
 inet_interfaces = all
+
+home_mailbox = Mail/
+smtpd_recipient_restrictions = permit_mynetworks reject_unauth_destination permit_sasl_authenticated
+smtpd_sasl_type = dovecot
+smtpd_sasl_path = private/auth
+smtp_helo_name = my.reverse.dns.name.com

At this point, also make sure that the parameters smtpd_tls_cert_file and smtpd_tls_key_file point to the right certificate and private key file. So either change these values or replace the content of /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key. The home_mailbox parameter sets the default path for incoming mail. Since there is no leading slash, this puts mail into $HOME/Mail for each user. The trailing slash is important as it specifies qmail-style delivery'' which means maildir. The default of the smtpd_recipient_restrictions parameter is permit_mynetworks reject_unauth_destination so this just adds the permit_sasl_authenticated option. This is necessary to allow users to send email when they successfully verified their login through dovecot. The dovecot login verification is activated through the smtpd_sasl_type and smtpd_sasl_path parameters. I found it necessary to set the smtp_helo_name parameter to the reverse DNS of my server. This was necessary because many other email servers would only accept email from a server with a valid reverse DNS entry. My hosting provider charges USD 7.50 per month to change the default reverse DNS name, so the easy solution is, to instead just adjust the name announced in the SMTP helo.

/etc/postfix/master.cf The file master.cf is used to enable the submission service. The following diff just removes the comment character from the appropriate section.

@@ -13,12 +13,12 @@
 #smtpd     pass  -       -       -       -       -       smtpd
 #dnsblog   unix  -       -       -       -       0       dnsblog
 #tlsproxy  unix  -       -       -       -       0       tlsproxy
-#submission inet n       -       -       -       -       smtpd
-#  -o syslog_name=postfix/submission
-#  -o smtpd_tls_security_level=encrypt
-#  -o smtpd_sasl_auth_enable=yes
-#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
-#  -o milter_macro_daemon_name=ORIGINATING
+submission inet n       -       -       -       -       smtpd
+  -o syslog_name=postfix/submission
+  -o smtpd_tls_security_level=encrypt
+  -o smtpd_sasl_auth_enable=yes
+  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
+  -o milter_macro_daemon_name=ORIGINATING
 #smtps     inet  n       -       -       -       -       smtpd
 #  -o syslog_name=postfix/smtps
 #  -o smtpd_tls_wrappermode=yes

Dovecot Since above configuration changes made postfix store email in a different location and format than the default, dovecot has to be informed about these changes as well. This is done in `/etc/dovecot/conf.d/10-mail.conf`. The second configuration change enables postfix to authenticate users through dovecot in `/etc/dovecot/conf.d/10-master.conf`. For SSL one should look into `/etc/dovecot/conf.d/10-ssl.conf` and either adapt the parameters `ssl_cert` and `ssl_key` or store the correct certificate and private key in `/etc/dovecot/dovecot.pem` and `/etc/dovecot/private/dovecot.pem`, respectively. The `dovecot-core` package (which `dovecot-imapd` depends on) ships tons of documentation. The file `/usr/share/doc/dovecot-core/dovecot/documentation.txt.gz` gives an overview of what resources are available. The path `/usr/share/doc/dovecot-core/dovecot/wiki` contains a snapshot of the dovecot wiki at http://wiki2.dovecot.org/. The example configurations seem to be the same files as in `/etc/` which are already well commented.

/etc/dovecot/conf.d/10-mail.conf The following diff changes the default email location in /var/mail to a maildir in ~/Mail as configured for postfix above.

@@ -27,7 +27,7 @@
 #
 # <doc/wiki/MailLocation.txt>
 #
-mail_location = mbox:~/mail:INBOX=/var/mail/%u
+mail_location = maildir:~/Mail
 
 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.

/etc/dovecot/conf.d/10-master.conf And this enables the authentication socket for postfix:

@@ -93,9 +93,11 @@
    
 
   # Postfix smtp-auth
-  #unix_listener /var/spool/postfix/private/auth  
-  #  mode = 0666
-  # 
+  unix_listener /var/spool/postfix/private/auth  
+    mode = 0660
+    user = postfix
+    group = postfix
+   
 
   # Auth process is run as this user.
   #user = $default_internal_user

Aliases Now Email will automatically put into the '~/Mail' directory of the receiver. So a user has to be created for whom one wants to receive mail...

$ adduser josch

...and any aliases for it to be configured in /etc/aliases.

@@ -1,2 +1,4 @@
-# See man 5 aliases for format
-postmaster:    root
+root:       josch
+postmaster: josch
+hostmaster: josch
+webmaster:  josch

After editing /etc/aliases, the command

$ newaliases

has to be run. More can be read in the aliases(5) man page.

Finishing up Everything is done and now postfix and dovecot have to be informed about the changes. There are many ways to do that. Either restart the services, reboot or just do:

$ postfix reload
$ doveadm reload

Have fun!

28 September 2014

Ean Schuessler: RoboJuggy at JavaOne

A few months ago I was showing my friend Bruno Souza the work I had been doing with my childhood friend and robotics genius, David Hanson. I had been watching what David was going through in his process of creating life-like robots with the limited industrial software available for motor control. I had suggested to David that binding motors to Blender control structures was a genuinely viable possibility. David talked with his forward looking CEO, Jong Lee, and they were gracious enough to invite me to Hong Kong to make this exciting idea a reality. Working closely the HRI team (Vytas, Gabrielos, Fabien and Davide) with David s friend and collaborators at OpenCog (Ben Goertzel, Mandeep, David, Jamie, Alex and Samuel) a month long creative hack-fest yielded pretty amazing results. Bruno is an avid puppeteer, a global organizer of java user groups and creator of Juggy the Java Finch, mascot of Java users and user groups everywhere. We started talking about how cool it would be to have a robot version of Juggy. When I was in China I had spent a little time playing with Mark Tilden s RSMedia and various versions of David s hobby servo based emotive heads. Bruno and I did a little research into the ROS Java bindings for the Robot Operating System and decided that if we could make that part of the picture we had a great and fun idea for a JavaOne talk. Hunting and gathering I tracked down a fairly priced RSMedia in Alaska, Bruno put a pair of rubber Juggy puppet heads in the mail and we were on our way.
We had decided that we wanted RoboJuggy to be able to run about untethered and the new RaspberryPi B+ seemed like the perfect low power brain to make that happen. I like the Debian based Raspbian distributions but had lately started using the netinst Pi images. These get your Pi up and running in about 15 minutes with a nicely minimalistic install instead of a pile of dependencies you probably don t need. I d recommend anyone interested I m duplicating our work to stay their journey there: Raspbian UA Net Installer Robots seem like an embedded application but ROS only ships packages for Ubuntu. I was pleasantly surprised that there are very good instructions for building ROS from source on the Pi. I ended up following these instructions: Setting up ROS Hydro on the Raspberry Pi Building from source means that all your install ends up being isolated (in ROS speak) and your file locations and build instructions end up being subtly current. As explained in the linked article, this process is also very time consuming. One thing I would recommend once you get past this step is to use the UNIX dd command to back up your entire SD card to a desktop. This way if you make a mess of things in later steps you can restore your install to a pristine Raspbian+ROS install. If your SD drive was on /dev/sdb you might use something like this to do the job:

sudo dd bs=4M if=/dev/sdb   gzip > /home/your_username/image date +%d%m%y .gz

Getting Java in the mix Once you have your Pi all set up with minimal Raspbian and ROS you are going to want a Java VM. The Pi runs the ARM CPU so you need the corresponding version of Java. I tried getting things going initially with OpenJDK and I had some issues with that. I will work on resolving that in the future because I would like to have a 100% Free Software kit for this but since this was for JavaOne I also wanted JDK8, which isn t available in Debian yet. So, I downloaded the Oracle JDK8 package for ARM. Java 8 JDK for ARM At this point you are ready to start installing the ROS Java packages. I m pretty sure the way I did this initially is wrong but I was trying to reconcile the two install procedures for ROS Java and ROS Hydro for Raspberry Pi. I started by following these directions for ROS Java but with a few exceptions (you have to click the install from source link in the page to see the right stuff: Installing ROS Java on Hydro Now these instructions are good but this is a Pi running Debian and not an Ubuntu install. You won t run the apt-get package commands because those tools were already installed in your earlier steps. Also, this creates its own workspace and we really want these packages all in one workspace. You can apparently chain workspaces in ROS but I didn t understand this well enough to get it working so what I did was this:

> mkdir -p ~/rosjava 
> wstool init -j4 ~/rosjava/src https://raw.github.com/rosjava/rosjava/hydro/rosjava.rosinstall
> source ~/ros_catkin_ws/install_isolated/setup.bash > cd ~/rosjava # Make sure we've got all rosdeps and msg packages.
> rosdep update 
> rosdep install --from-paths src -i -y

and then copied the sources installed into ~/rosjava/src into my main ~/ros_catkin_ws/src. Once those were copied over I was able to run a standard build.

> catkin_make_isolated --install

Like the main ROS install this process will take a little while. The Java gradle builds take an especially long time. One thing I would recommend to speed up your workflow is to have an x86 Debian install (native desktop, QEMU instance, docker, whatever) and do these same build from source installs there. This will let you try your steps out on a much faster system before you try them out in the Pi. That can be a big time saver. Putting together the pieces Around this time my RSMedia had finally showed up from Alaska. At first I thought I had a broken unit because it would power up, complain about not passing system tests and then shut back down. It turns out that if you just put the D batteries in and miss the four AAs that it will kind of pretend to be working so watch for that mistake. Here is a picture of the RSMedia when it first came out of the box:

Other parts were starting to roll in as well. The rubber puppet heads had made their way through Brazilian customs and my Pololu Mini Maestro 24 had also shown up as well as the my servo motors and pan and tilt camera rig. I had previously bought a set of 10 motors for goofing around so I bought the pan and tilt rig by itself for about $5(!) but you can buy a complete set for around $25 from a number of EBay stores. Complete pan and tilt rig with motors for $25 A bit more about the Pololu. This astonishing little motor controller costs about $25 and gives you control of 24 motors with an easy to use and high level serial API. It is probably also possible to control these servos directly from the Pi and eliminate this board but that will be genuinely difficult because of the real-time timing issues. For $25 this thing is a real gem and you won t regret buying it. Now it was time to start dissecting the RSMedia and getting control of its brain. Unfortunately a lot of great information about the RSMedia has floated away since it was in its heyday 5 years ago but there is still some solid information out there that we need to round up and preserve. A great resource is the SourceForge based website here at http://rsmediadevkit.sourceforge.net. That site has links to a number of useful sites. You will definitely want to check out their wiki. To disassemble the RSMedia I followed their instructions. I will say, it would be smart to take more pictures as you are going because they don t take as many as they should. I took pictures of each board and its associated connections as dismantled the unit and that helped me get things back together later. Another important note is that if all you want to do is solder onto the control board and not replace the head then its feasible to solder the board in place without completely disassembling the unit. Here are some photos of the dis-assembly:

Now I also had to start adjusting the puppet head, building an armature for the motors to control it and hooking it into the robot. I need to take some more photos of the actual armature. I like to use cardboard for this kind of stuff because it is so fast to work with and relatively strong. One trick I have also learned about cardboard is that if you get something going with it and you need it to be a little more production strength you can paint it down with fiberglass resin from your local auto store. Once it dries it becomes incredibly tough because it soaks through the fibers of the cardboard and hardens around them. You will want to do this in a well ventilated area but its a great way to build super tough prototypes. Another prototyping trick I can suggest is using a combination of Velcro and zipties to hook things together. The result is surprisingly strong and still easy to take apart if things aren t working out. Velcro self-adhesive pads stick to rubber like magic and that is actually how I hooked the jaw servo onto the mask. You can see me torturing its first initial connection here: Since the puppet head had come all the way from Brazil I decided to cook some chicken hearts in the churrascaria style while I worked on them in the garage. This may sound gross but I m telling you, you need to try it! I soaked mine in soy sauce, Sriracha and chinese cooking wine. Delicious but I digress.

As I was eating my chicken hearts I was also connecting the pan and tilt armature onto the puppet s jaw and eye assembly. It took me most of the evening to get all this going but by about one in the morning things were starting to look good! I only had a few days left to hack things together before JavaOne and things were starting to get tight. I had so much to do and had also started to run into some nasty surprises with the ROS Java control software. It turns out that ROS Java is less than friendly with ROS message structures that are not built in . I had tried to follow the provided instructions but was not (and still have not) been able to get that working. Using unofficial messages with ROS Java I still needed to get control of the RSMedia. Doing that required the delicate operation of soldering to its control board. On the board there are a set of pins that provide a serial interface to the ARM based embedded Linux computer that controls the robot. To do that I followed these excellent instructions: Connecting to the RSMedia Linux Console Port After some sweaty time bent over a magnifying glass I had success:

I had previously purchased the USB-TTL232 accessory described in the article from Dallas awesome Tanner Electronics store in Dallas. If you are a geek I would recommend that you go there and say hi to its proprietor (and walking encyclopedia of electronics knowledge) Jim Tanner. It was very gratifying when I started a copy of minicom, set it to 115200, N, 8, 1, plugged in the serial widget to the RSMedia and booted it up. I was greeted with a clearly recognizable Linux startup and console prompt. At first I thought I had done something wrong because I couldn t get it to respond to commands but I quickly realized I had flow control turned on. Once turned off I was able to navigate around the file system, execute commands and have some fun. A little research and I found this useful resource which let me get all kinds of body movements going: A collection of useful commands for the RSMedia At this point, I had a usable set of controls for the body as well as the neck armature. I had a controller running the industry s latest and greatest robotics framework that could run on the RSMedia without being tethered to power and I had most of a connection to Java going. Now I just had to get all those pieces working together. The only problem is that time was running out and I only had a couple of days until my talk and still had to pack and square things away at work. The last day was spent doing things that I wouldn t be able to do on the road. My brother Erik (and fantastic artist) came over to help paint up the juggy head and fix the eyeball armature. He used a mix of oil paint, rubber cement which stuck to the mask beautifully. I bought battery packs for the USB Pi power and the 6v motor control and integrated them into a box that could sit below the neck armature. I fixed up a cloth neck sleeve that could cover everything. Luckily during all this my beautiful and ever so supportive girlfriend Becca had helped me get packed or I probably wouldn t have made it out the door. Welcome to San Francisco THIS ARTICLE IS STILL BEING WRITTEN

6 September 2014

Russ Allbery: Accumulated hauls

I haven't made one of these in a long time, so I have some catching from random purchases to do, which includes a (repurposed) nice parting gift from my previous employer and a trip to Powell's since I was in the area for DebConf14. This also includes the contents of the Hugo voter's packet, which contained a wide variety of random stuff even if some of the novels were represented only by excerpts. John Joseph Adams (ed.) The Mad Scientist's Guide to World Domination (sff anthology)
Roger McBride Allen The Ring of Charon (sff)
Roger McBride Allen The Shattered Sphere (sff)
Iain M. Banks The Hydrogen Sonata (sff)
Julian Barnes The Sense of an Ending (mainstream)
M. David Blake (ed.) 2014 Campbellian Anthology (sff anthology)
Algis Budrys Benchmarks Continued (non-fiction)
Algis Budrys Benchmarks Revisited (non-fiction)
Algis Budrys Benchmarks Concluded (non-fiction)
Edgar Rice Burroughs Carson of Venus (sff)
Wesley Chu The Lives of Tao (sff)
Ernest Cline Ready Player One (sff)
Larry Correia Hard Magic (sff)
Larry Correia Spellbound (sff)
Larry Correia Warbound (sff)
Sigrid Ellis & Michael Damien Thomas (ed.) Queer Chicks Dig Time Lords (non-fiction)
Neil Gaiman The Ocean at the End of the Lane (sff)
Max Gladstone Three Parts Dead (sff)
Max Gladstone Two Serpents Rise (sff)
S.L. Huang Zero Sum Game (sff)
Robert Jordan & Brandon Sanderson The Wheel of Time (sff)
Drew Karpyshyn Mass Effect: Revelation (sff)
Justin Landon & Jared Shurin (ed.) Speculative Fiction 2012 (non-fiction)
John J. Lumpkin Through Struggle, the Stars (sff)
L. David Marquet Turn the Ship Around! (non-fiction)
George R.R. Martin & Raya Golden Meathouse Man (graphic novel)
Ramez Naam Nexus (sff)
Eiichiro Oda One Piece Volume 1 (manga)
Eiichiro Oda One Piece Volume 2 (manga)
Eiichiro Oda One Piece Volume 3 (manga)
Eiichiro Oda One Piece Volume 4 (manga)
Alexei Panshin New Celebrations (sff)
K.J. Parker Devices and Desires (sff)
K.J. Parker Evil for Evil (sff)
Sofia Samatar A Stranger in Olondria (sff)
John Scalzi The Human Division (sff)
Jonathan Straham (ed.) Fearsome Journeys (sff anthology)
Vernor Vinge The Children of the Sky (sff)
Brian Wood & Becky Cloonan Demo (graphic novel)
Charles Yu How to Live Safely in a Science Fictional Universe (sff) A whole bunch of this is from the Hugo voter's packet, and since the Hugos are over, much of that probably won't get prioritized. (I was very happy with the results of the voting, though.) Other than that, it's a very random collection of stuff, including a few things that I picked up based on James Nicoll's reviews. Now that I have a daily train commute, I should pick up the pace of reading, and as long as I can find enough time in my schedule to also write reviews, hopefully there will be more content in this blog shortly.

5 June 2014

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was four months ago and about several incremental updates. This post will be of similar nature. The most interesting news is probably the additional data that bootstrap.debian.net now provides. This is listed in the next section. All subsequent sections then list the changes under the hood that made the additions to bootstrap.debian.net possible.

bootstrap.debian.net The bootstrap.debian.net service used to have botch as a git submodule but now runs botch from its Debian package. This at least proves that the botch Debian package is mature enough to do useful stuff with it. In addition to the bootstrapping results by architecture, bootstrap.debian.net now also hosts the following additional services:

History of graph size shows how the dependency graph developed over time for a normal self-contained repository plus for both minimizing strategies, updated every five days

Crossbuild dependency satisfaction gives an overview of reasons why the crossbuild dependency situation cannot yet be analyzed with bug numbers where applicable

Crossbuild order modifies the metadata of a repository so that the crossbuild dependency situation can be analyzed and then outputs an overview page as it is done for every architecture on the main page

Source package importance calculates the port metric for source packages on a daily basis

Further improvements concern how dependency cycles are now presented in the html overviews. While before, vertices in a cycle where separated by commas as if they were simple package lists, vertices are now connected by unicode arrows. Dashed arrows indicate build dependencies while solid arrows indicate builds-from relationships. For what it's worth, installation set vertices now contain their installation set in their `title` attribute.

Debian package Botch has long depended on features of an unreleased version of `dose3` which in turn depended on an unrelease version of `libcudf`. Both projects have recently made new releases so that I was now able to drop the `dose3` git submodule and rely on the host system's `dose3` version instead. This also made it possible to create a Debian package of botch which currently sits at Debian mentors. Writing the package also finally made me create a usable `install` target in the `Makefile` as well as adding stubs for the manpages of the 44 applications that botch currently ships. The actual content of these manpages still has to be written. The only documentation botch currently ships in the `botch-doc` package is an offline version of the wiki on gitorious. The new page ExamplesGraphs even includes pictures.

Cross By default, botch analyzes the native bootstrapping phase. That is, assume that the initial set of `Essential:yes` and `build-essential` packages magically exists and find out how to bootstrap the rest from there through native compilation. But part of the bootstrapping problem is also to create the set of `Essential:yes` and `build-essential` packages from nothing via cross compilation. Botch is unable to analyze the cross phase because too many packages cannot satisfy their crossbuild dependencies due to multiarch conflicts. This problem is only about the dependency metadata and not about whether a given source package actually crosscompiles fine in practice. Helmut Grohne has done great work with rebootstrap which is regularly run by jenkins.debian.net. He convinced me that we need an overview of what packages are blocking the analysis of the cross case and that it was useful to have a crossbuild order even if that was a fake order just to have a rough overview of the current situation in Debian Sid. I wrote a couple of scripts which would run `dose-builddebcheck` on a repository, analyze which packages fail to satisfy their crossbuild dependencies and why, fix those cases by adjusting package metadata accordingly and repeat until all relevant source packages satisfy their crossbuild dependencies. The result of this can then be used to identify the packages that need to be modified as well as to generate a crossbuild order. The fixes to the metadata are done in an automatic fashion and do not necessarily reflect the real fix that would solve the problem. Nevertheless, I ended up agreeing that it is better to have a slightly wrong overview than no overview at all.

Minimizing the dependency graph size Installation sets in the dependency graph are calculated independent from each other. If two binary packages provide `A`, then dependencies on `A` in different installation sets might choose different binary packages as providers of `A`. The same holds for disjunctive dependencies. If a package depends on `A C` and another package depends on `C A` then there is no coordination to choose `C` so to minimize the overall amount of vertices in the graph. I implemented two methods to minimize the impact of cases where the dependency solver has multiple options to satisfy a dependency through `Provides` and dependency disjunctions. The first method is inspired by Helmut Grohne. An algorithm goes through all disjunctive binary dependencies and removes all virtual packages, leaving only real packages. Of the remaining real packages, the first one is selected. For build dependencies, the algorithm drops all but the first package in every disjunction. This is also what sbuild does. Unfortunately this solution produces an unsatisfiable dependency situation in most cases. This is because oftentimes it is necessary to select the virtual disjunctive dependency because of a conflict relationship introduced by another package. The second method involves `aspcud`, a cudf solver which can optimize a solution by a criteria. This solution is based on an idea by Pietro Abate who implemented the basis for this idea back in 2012. In contrast to a usual cudf problem, binary packages now also depend on the source packages they build from. If we now ask `aspcud` to find an installation set for one of the base source packages (I chose `src:build-essential`) then it will return an installation set that includes source packages. As an optimization criteria the number of source packages in the installation set is minimized. This solution would be flawless if there were no conflicts between binary packages. Due to conflicts not all binary packages that must be coinstallable for this strategy to work can be coinstalled. The quick and dirty solution is to remove all conflicts before passing the cudf universe to `aspcud`. But this also means that the solution does sometimes not work in practice.

Test cases Botch now finally has a `test` target in its `Makefile`. The `test` target tests two code paths of the `native.sh` script and the `cross.sh` script. Running these two scripts covers testing most parts of botch. Given that I did lots of refactoring in the past weeks, the test cases greatly helped to assure that I didnt break anything in the process. I also added autopkgtests to the Debian packaging which test the same things as the `test` target but naturally run the installed version of botch instead. The autopkgtests were a great help in weeding out some lasts bugs which made botch depend on being executed from its source directory.

Python 3 Reading the suggestions in the Debian python policy I evaluated the possibility to use Python 3 for the Python scripts in botch. While I was at it I added transparent decompression for gzip, bz2 and xz based on the file magic, replaced python-apt with python-debian because of bug#748922 and added `argparse` argument parsing to all scripts. Unfortunately I had to find out that Python 3 support does not yet seem to be possible for botch for the following reasons:

no soap module for Python 3 in Debian (needed for bts access)

hash randomization is turned on by default in Python 3 and therefore the graph output of networkx is not deterministic anymore (bug#749710)

Thus I settled for changing the code such that it would be compatible with Python 2 as well as with Python 3. Because of the changed string handling and `sys.stdout` properties in Python 3 this proved to be tricky. On the other hand this showed me bugs in my code where I was wrongly relying on deterministic dictionary key traversal.

13 December 2013

Keith Packard: xserver-warnings

Cleaning up X server warnings So I was sitting in the Narita airport with a couple of other free software developers merging X server patches. One of the developers was looking over my shoulder while the X server was building and casually commented on the number of warnings generated by the compiler. I felt like I had invited someone into my house without cleaning for months embarrassed and ashamed that we d let the code devolve into this state. Of course, we ve got excuses the X server code base is one of the oldest pieces of regularly used free software in existence. It was started before ANSI-C was codified. No function prototypes, no const , no void * , no enums or stdint.h. There may be a few developers out there who remember those days (fondly, of course), but I think most of us are glad that our favorite systems language has gained a lot of compile-time checking in the last 25 years. We ve spent time in the past adding function prototypes and cleaning up other warnings, but there was never a point at which the X server compiled without any warnings. More recently, we ve added a pile of new warning flags when compiling the X server which only served to increase the number of warnings dramatically. The current situation With the master branch of the X server and released versions of the dependencies, we generate 1047 warnings in the default build. -Wcast-qual considered chatty The GCC flag, -Wcast-qual, complains when you cast a pointer and change the const qualifier status. A very common thing for the X server to do is declare pointers as const to mark them as immutable once assigned. Often, the relevant data is actually constructed once at startup in allocated memory and stored to the data structure. During server reset, that needs to be freed, but free doesn t take a const pointer, so we cast to (void *), which -Wcast-qual then complains about. Loudly. Of the 1047 warnings, 380 of them are generated by this one warning flag. I ve gone ahead and just disabled it in util/macros for now. String constants are a pain The X server uses string constants to initialize defaults for font paths, configuration options, font names along with a host of other things. These end up getting stored in variables that can also take allocated storage. I ve gone ahead and declared the relevant objects as const and then fixed the code to suit. I don t have a count of the number of warnings these changes fixed; they were scattered across dozens of X server directories, and I was fixing one directory at a time, but probably more than half of the remaining warnings were of this form. And a host of other warnings Fixing the rest of the warnings was mostly a matter of stepping through them one at a time and actually adjusting the code. Shadowed declarations, unused values, redundant declarations and missing printf attributes were probably the bulk of them though. Changes to external modules Instead of just hacking the X server code, I ve created patches for other modules where necessary to fix the problems in the right place.

proto/fontsproto. Declares FontPathElement names as const char *
mesa/drm. Adds printf attribute to the debug_print function
util/macros. Removes -Wcast-qual from the default warning set.

Getting the bits In case it wasn t clear, the X server build now generates zero warnings on my machine. I m hoping that this will also be true for other people. Patches are available at:

xserver - git://people.freedesktop.org/~keithp/xserver warning-fixes
fontsproto - git://people.freedesktop.org/~keithp/fontsproto fontsproto-next
mesa/drm - git://people.freedesktop.org/~keithp/drm warning-fixes
util/macros - already upstream on master

Keeping our house clean Of course, these patches are all waiting until 1.15 ships so that we don t accidentally break something important. However, once they re merged, I ll be bouncing any patches which generate warnings on my system, and if other people find warnings when they build, I ll expect them to send patches as well. Now to go collect the tea cups in my office and get them washed along with the breakfast dishes so I won t be embarrassed if some free software developers show up for lunch today.

24 October 2013

Russ Allbery: Review: Applied Cryptography

Review: Applied Cryptography, by Bruce Schneier

Publisher:	John Wiley & Sons
Copyright:	1996
ISBN:	0-471-11709-9
Format:	Trade paperback
Pages:	621

This is the next entry in the series of Russ reading books that he bought years ago and never got around to reading. Thankfully, this time, the book has aged somewhat better. This review is for the second edition of Applied Cryptography, published in 1996. Given how important computer security has become, and how central cryptography is to computer security, one might think that the passage of 17 years would make a book effectively obsolete. This turns out not to be the case. Yes, Rijndael (the current AES standard and the most widely-used block cipher), Camellia (the up-and-comer in the block cipher world), and the SHA-2 hash postdate this book and aren't discussed. Yes, there have been some further developments in elliptic-curve public-key cryptography. And yes, much of the political information in this book, as well as the patent situation for public-key cryptosystems, is now mostly of historical interest. But a surprising amount of this book still applies directly. Partly that's because much of Applied Cryptography is focused on general principles, classes of algorithms, and cryptographic protocols, and those do not change quickly. A rather amazing number of techniques and protocols still in widespread use today originated in the 1970s. Block ciphers, stream ciphers, cipher modes, public-key systems, signature systems, hash functions, key exchange, secret splitting, key management, and other, similar topics have not changed substantially. And even in the area of specific algorithms, DES is still in use (unfortunately) and its analysis is valuable for understanding any block cipher. RC4, Triple DES, CAST, RSA, MD4, MD5, and SHA-1 are still part of the cryptography landscape, and Schneier was already warning against using MD5 in 1996. The source code included in this book has, thankfully, been made obsolete by widely-available, high-quality free implementations of all common ciphers, available for any modern operating system. But the math, the explanations, and much of the cryptanalysis is still quite relevant. While it contains a comprehensive introduction to cryptographic concepts, Applied Cryptography is structured more like a reference manual than a tutorial. The first two chapters provide some foundations. Then, a wide variety of protocols, from the common to the completely esoteric, are discussed across the subsequent four chapters. There are four chapters on the basics of ciphers, another chapter on the mathematics, and then the longest part of the book: twelve chapters that provide a tour of all major and several interesting minor protocols that existed in 1996. These are organized into block ciphers, stream ciphers (and pseudo-random number generators), hash functions, and public-key systems. An entire chapter is devoted to the history and analysis of DES, and I think that was my favorite chapter of the book. Not only is it worth understanding how DES works, but it also provides a comprehensive introduction to the complexities of block cipher design, the politics of interactions with the NSA, and a fascinating history of early computer cryptosystems. Finally, Applied Cryptography closes with chapters on real-world systems and on politics. The politics section is mostly a historical curiosity, as is much of the chapter on real-world systems. Schneier discusses some of the PKCS and X.509 protocols here but doesn't use SSL as one of his examples despite its specification predating this book (perhaps the most glaring omission of the book). But he does discuss Kerberos (which is of personal interest to me), and it was useful to see it analyzed in the broader context of this sort of book. Those who read Schneier's blog regularly, or who have read any of his other books, will know that he's concise, incisive, and very readable, even with difficult material. I think he does err on the side of compactness this is not a book that handholds the reader or explains something more than once but there's a lot of ground to cover, so I can't really blame him. Expect to have to read the more difficult parts several times, and (at least on a first read) expect to just skip past some of the most complex sections as probably not worth the effort. But I think Schneier does a great job structuring the book so that one comes away with a high-level impression and overview sufficient to make informed guesses about capabilities and relative difficulty even if one doesn't have the patience to work through the details. I suspect most readers will want to skim at least part of this book. Unless you're willing to do a lot of background reading to understand the math, or a lot of careful study to follow the detailed descriptions and flaws in often-obscure algorithms, the algorithm discussions start to blend together. I found myself skipping a lot of the math and focusing on the basic descriptions and the cryptanalysis, particularly for the obscure ciphers. Many of the algorithms are also highly opaque and of obscure benefit; I doubt I will ever care about blind signatures or oblivious transfer, and I struggled to wrap my mind around the details of bit commitment (although it's apparently important in zero knowledge proofs, which in turn are relevant to authentication). But this is not the sort of book where you have to read every word. It's very well-structured, provides clear resynchronization points for readers who have had enough of a particular topic, and marks the topics that are more obscure or of dubious general usefulness. The most useful parts of this book, in my opinion, are the basic conceptual introductions to each class of cryptographic algorithm or protocol, and those chapters are excellent. I've been working in applied computer security, at the application and deployment level, for years, and this is the first time that I've felt like I really understood the difference between a block cipher and a stream cipher, or the implications of block cipher mode selection. Applied Cryptography is also a fascinating exercise in adjusting one's intuition for where the complexity lies. It's a memorable experience to move from the chapters on block ciphers, full of S-boxes and complex mixing and magic numbers, into the far more straightforward mathematical analysis of stream ciphers. Or the mathematical simplicity of RSA (although quite a bit of complexity is lurking on the cryptanalysis side, much of which is only touched on here). Of more historical interest, but still quite impressive, is that Applied Cryptography doubles as a comprehensive literature review. Not only does it cover nearly every algorithm in use at the time, it discusses the cryptography literature on nearly every topic with brief summaries of results and lines of investigation. The references section is a stunning 66 pages of small print featuring 1,653 references, and those are mentioned and put in context throughout the book. I'm not personally interested in chasing those lines of research further, and of course 17 years means there are many new papers that are as important to read today, but it's obvious that this book would have been an amazing and invaluable map to the research territory when it was first published. There are now newer books on this topic that you should consider if you're purchasing a book today. Cryptography Engineering in particular is in my to-read pile, and I'm interested to see how much of this book it could replace, although I believe it's lighter on the mathematical analysis and details. But I think Applied Cryptography still has a place on the bookshelf of any computer security professional. Schneier is succinct, detailed, straightforward, and comprehensive, at least for 1996, and I know I'll be reaching for this book the next time I forget the details of CFB mode and then get hopelessly confused by the Wikipedia article on the topic. Rating: 8 out of 10

5 October 2013

Russ Allbery: Miscellaneous haul

Despite the fact that I've had very little time to either read or write reviews, I haven't stopped buying a few books here and there. And, of course, Powell's Indiespensable keeps coming. Matt Bell In the House Upon the Dirt Between the Lake and the Woods (mainstream)
Steven Brust Orca (sff)
Steven Brust Dragon (sff)
John W. Campbell Collected Editorials from Analog (non-fiction)
Jane Fletcher The Walls of Westernfort (sff)
Mira Grant How Green This Land, How Blu This Sea (sff)
Tate Hallaway Tall, Dark, & Dead (sff)
Diana Wynne Jones Howl's Moving Castle (sff)
Rebecca Lee Bobcat and Other Stories (mainstream)
Victor Lombardi Why We Fail (non-fiction)
George R.R. Martin & Gardner Dozois (ed.) Songs of Love and Death (sff anthology)
Seanan McGuire In Sea-Salt Tears (sff)
Seanan McGuire Late Eclipses (sff)
Justina Robson Selling Out (sff)
John Scalzi Muse of Fire (sff)
Mark Slouka Brewster (mainstream)
Tor.com (ed.) The Stories (sff anthology)
Carrie Vaughn Kitty and the Midnight Hour (sff) This is a wide variety of random stuff, including some stuff that was free (the Tor.com anthology, which is every piece of short fiction they've published and is about 4000 pages, the Campbell editorials from the Internet Archive) and a lot of stuff that was on sale. As well as another two shipments of Indiespensible. The Brust books I've already read and are already in the family, but I hooked a friend of mine on the Vlad Taltos series, and those were the only two that I both didn't have and that weren't available on Kindle. So I picked them up while he was visiting so that he could read the whole series.

Next.

Previous.